Project 3 - Example Main Script

Step 0: specify directories.

Set the working directory to the image folder. Specify the training and the testing set. For data without an independent test/validation set, you need to create your own testing data by random subsampling. In order to obain reproducible results, set.seed() whenever randomization is used.

set.seed(2018)
# setwd("Fall2018-Proj3-Sec1-grp8/") 
# here replace it with your own path or manually set it in RStudio to where this rmd file is located. 
# use relative path for reproducibility

Provide directories for training images. Low-resolution (LR) image set and High-resolution (HR) image set will be in different subfolders.

train_dir <- "../data/train_set/" # This will be modified for different data sets.
train_LR_dir <- paste(train_dir, "LR/", sep="")
train_HR_dir <- paste(train_dir, "HR/", sep="")
train_label_path <- paste(train_dir, "label.csv", sep="") 

Step 1: set up controls for evaluation experiments.

In this chunk, we have a set of controls for the evaluation experiments.

• (T/F) cross-validation on the training set
• (number) K, the number of CV folds
• (T/F) process features for training set
• (T/F) run evaluation on an independent test set
• (T/F) process features for test set

run.cv=TRUE # run cross-validation on the training set
K <- 5  # number of CV folds
run.feature.train=TRUE # process features for training set
run.test=TRUE # run evaluation on an independent test set
run.feature.test=TRUE # process features for test set

Using cross-validation or independent test set evaluation, we compare the performance of models with different specifications. In this example, we use GBM with different depth. In the following chunk, we list, in a vector, setups (in this case, depth) corresponding to models that we will compare. In your project, you might compare very different classifiers. You can assign them numerical IDs and labels specific to your project.

model_values <- seq(3, 11, 2)
model_labels = paste("GBM with depth =", model_values)

Step 2: import training images class labels.

We provide extra information of image label: car (0), flower (1), market (2). These labels are not necessary for your model.

extra_label <- read.csv(train_label_path, colClasses=c("NULL", NA, NA))

Step 3: construct features and responses

feature.R should be the wrapper for all your feature engineering functions and options. The function feature( ) should have options that correspond to different scenarios for your project and produces an R object that contains features and responses that are required by all the models you are going to evaluate later. + feature.R + Input: a path for low-resolution images. + Input: a path for high-resolution images. + Output: an RData file that contains extracted features and corresponding responses

dat_train$feature

, , 1

                 [,1]         [,2]         [,3]         [,4]         [,5]         [,6]
    [1,] -0.015686275 -0.235294118 -0.047058824 -0.192156863  0.007843137 -0.050980392
    [2,]  0.200000000  0.266666667  0.007843137 -0.086274510 -0.192156863  0.035294118
    [3,]  0.023529412  0.031372549  0.015686275  0.023529412  0.019607843  0.019607843
    [4,] -0.133333333 -0.164705882 -0.149019608 -0.074509804 -0.117647059  0.047058824
    [5,]  0.043137255 -0.152941176 -0.345098039  0.015686275 -0.156862745 -0.121568627
    [6,]  0.447058824  0.431372549  0.439215686  0.203921569  0.003921569 -0.274509804
    [7,] -0.184313725 -0.168627451 -0.109803922 -0.054901961 -0.050980392 -0.058823529
    [8,] -0.098039216 -0.137254902 -0.105882353 -0.121568627  0.019607843  0.192156863
    [9,]  0.000000000 -0.054901961 -0.054901961 -0.031372549 -0.039215686 -0.050980392
   [10,]  0.003921569  0.109803922 -0.011764706  0.019607843 -0.003921569 -0.105882353
   [11,] -0.003921569  0.031372549  0.003921569  0.082352941  0.019607843  0.082352941
   [12,] -0.039215686 -0.027450980 -0.023529412  0.003921569 -0.015686275  0.007843137
   [13,]  0.145098039  0.270588235  0.254901961  0.054901961  0.094117647  0.019607843
   [14,]  0.149019608  0.011764706  0.062745098  0.172549020 -0.019607843  0.058823529
   [15,]  0.101960784  0.062745098  0.043137255 -0.031372549 -0.054901961 -0.133333333
   [16,]  0.231372549  0.113725490  0.011764706  0.117647059  0.035294118 -0.050980392
   [17,]  0.317647059  0.384313725  0.337254902 -0.011764706 -0.066666667  0.235294118
   [18,]  0.035294118  0.019607843 -0.015686275  0.011764706 -0.270588235  0.000000000
   [19,] -0.007843137 -0.027450980  0.007843137  0.003921569 -0.003921569  0.003921569
   [20,] -0.086274510 -0.047058824  0.101960784 -0.137254902 -0.184313725 -0.062745098
   [21,] -0.184313725  0.333333333  0.152941176 -0.098039216 -0.082352941  0.133333333
   [22,]  0.011764706  0.007843137  0.015686275  0.015686275 -0.019607843  0.086274510
   [23,]  0.023529412  0.007843137  0.043137255  0.054901961 -0.007843137  0.015686275
   [24,] -0.019607843 -0.007843137 -0.007843137 -0.003921569 -0.011764706  0.039215686
   [25,] -0.043137255 -0.031372549 -0.019607843 -0.019607843 -0.003921569 -0.015686275
   [26,] -0.125490196 -0.058823529 -0.039215686 -0.129411765  0.121568627 -0.023529412
   [27,] -0.035294118 -0.141176471 -0.098039216 -0.023529412 -0.015686275 -0.101960784
   [28,]  0.000000000  0.003921569  0.003921569  0.000000000 -0.023529412 -0.050980392
   [29,]  0.050980392  0.019607843 -0.027450980  0.007843137 -0.007843137 -0.007843137
   [30,] -0.047058824 -0.047058824 -0.094117647  0.000000000 -0.027450980 -0.007843137
   [31,] -0.039215686  0.023529412  0.027450980  0.003921569  0.019607843  0.050980392
   [32,] -0.011764706 -0.007843137 -0.298039216 -0.011764706 -0.015686275  0.023529412
   [33,]  0.109803922  0.027450980  0.023529412  0.066666667  0.074509804  0.243137255
   [34,] -0.015686275 -0.015686275 -0.007843137  0.003921569 -0.070588235 -0.039215686
   [35,]  0.117647059  0.062745098  0.035294118 -0.011764706  0.062745098 -0.003921569
   [36,] -0.027450980 -0.050980392 -0.125490196  0.003921569 -0.015686275  0.000000000
   [37,]  0.000000000 -0.015686275 -0.050980392 -0.019607843 -0.015686275  0.015686275
   [38,]  0.035294118  0.058823529  0.011764706 -0.019607843 -0.011764706  0.094117647
   [39,] -0.007843137  0.007843137  0.019607843  0.011764706  0.003921569 -0.039215686
   [40,] -0.113725490 -0.058823529 -0.027450980 -0.054901961 -0.011764706  0.074509804
   [41,] -0.011764706  0.003921569  0.007843137 -0.019607843 -0.007843137 -0.007843137
   [42,] -0.011764706  0.015686275  0.027450980 -0.015686275  0.007843137 -0.011764706
   [43,]  0.031372549 -0.003921569 -0.031372549 -0.082352941 -0.047058824 -0.298039216
   [44,]  0.054901961  0.050980392  0.035294118 -0.027450980 -0.054901961  0.039215686
   [45,] -0.207843137  0.058823529 -0.011764706 -0.156862745 -0.003921569 -0.109803922
   [46,] -0.062745098  0.007843137  0.007843137 -0.054901961  0.043137255  0.058823529
   [47,] -0.050980392 -0.086274510 -0.094117647 -0.047058824 -0.015686275 -0.070588235
   [48,]  0.003921569 -0.137254902  0.062745098  0.011764706  0.027450980 -0.078431373
   [49,] -0.019607843 -0.011764706 -0.015686275 -0.015686275  0.011764706 -0.035294118
   [50,]  0.027450980 -0.215686275 -0.109803922 -0.352941176 -0.125490196 -0.035294118
   [51,] -0.082352941 -0.039215686 -0.109803922 -0.137254902 -0.039215686 -0.172549020
   [52,]  0.243137255  0.219607843  0.145098039  0.219607843  0.184313725  0.086274510
   [53,]  0.003921569  0.000000000  0.039215686  0.011764706  0.003921569 -0.003921569
   [54,] -0.549019608 -0.384313725  0.019607843 -0.533333333  0.015686275 -0.074509804
   [55,] -0.054901961  0.000000000 -0.023529412 -0.047058824 -0.031372549  0.011764706
   [56,] -0.058823529  0.011764706 -0.078431373  0.086274510 -0.019607843  0.054901961
   [57,]  0.176470588 -0.007843137  0.149019608  0.235294118  0.117647059  0.227450980
   [58,] -0.039215686 -0.011764706 -0.047058824 -0.007843137 -0.070588235 -0.015686275
   [59,]  0.015686275  0.019607843  0.058823529  0.007843137  0.043137255  0.000000000
   [60,]  0.376470588  0.466666667  0.486274510 -0.011764706 -0.011764706 -0.078431373
   [61,]  0.105882353  0.094117647  0.086274510 -0.003921569  0.050980392 -0.047058824
   [62,]  0.007843137 -0.011764706 -0.007843137  0.015686275  0.027450980  0.000000000
   [63,]  0.062745098 -0.027450980  0.019607843  0.000000000  0.058823529  0.062745098
   [64,]  0.129411765  0.145098039  0.125490196 -0.027450980  0.000000000 -0.015686275
   [65,] -0.047058824 -0.011764706  0.062745098 -0.286274510  0.117647059 -0.066666667
   [66,] -0.019607843 -0.015686275 -0.015686275 -0.035294118 -0.011764706 -0.031372549
   [67,]  0.003921569  0.019607843  0.003921569  0.003921569  0.027450980 -0.003921569
   [68,] -0.011764706 -0.023529412  0.003921569  0.027450980 -0.007843137  0.007843137
   [69,]  0.019607843  0.011764706  0.019607843  0.031372549 -0.015686275  0.031372549
   [70,]  0.039215686 -0.031372549 -0.019607843  0.003921569 -0.203921569  0.090196078
   [71,]  0.019607843  0.015686275  0.058823529  0.007843137  0.043137255 -0.015686275
   [72,]  0.156862745  0.172549020  0.160784314 -0.011764706 -0.054901961 -0.105882353
   [73,]  0.658823529  0.596078431  0.121568627  0.592156863  0.670588235  0.047058824
   [74,]  0.074509804 -0.039215686  0.078431373  0.337254902  0.105882353  0.407843137
   [75,]  0.023529412  0.082352941  0.117647059  0.109803922  0.164705882  0.137254902
   [76,] -0.050980392  0.007843137 -0.003921569 -0.023529412  0.043137255 -0.058823529
   [77,] -0.062745098 -0.066666667 -0.023529412 -0.043137255  0.043137255  0.027450980
   [78,] -0.117647059  0.027450980  0.043137255  0.023529412  0.215686275  0.168627451
   [79,] -0.156862745 -0.376470588 -0.447058824 -0.215686275 -0.027450980 -0.482352941
   [80,] -0.250980392  0.054901961 -0.203921569 -0.137254902  0.031372549  0.125490196
   [81,] -0.121568627 -0.172549020 -0.156862745 -0.082352941 -0.164705882 -0.074509804
   [82,] -0.160784314 -0.113725490 -0.117647059  0.047058824 -0.066666667 -0.058823529
   [83,]  0.003921569  0.043137255  0.007843137  0.019607843 -0.003921569 -0.019607843
   [84,] -0.164705882  0.011764706  0.082352941 -0.031372549  0.031372549  0.027450980
   [85,]  0.054901961 -0.050980392  0.141176471  0.105882353  0.086274510  0.137254902
   [86,]  0.203921569 -0.086274510 -0.062745098  0.188235294  0.145098039 -0.129411765
   [87,]  0.117647059  0.145098039  0.164705882 -0.160784314  0.047058824 -0.223529412
   [88,]  0.011764706  0.027450980 -0.007843137 -0.023529412 -0.007843137  0.003921569
   [89,] -0.023529412  0.007843137  0.027450980  0.003921569  0.003921569  0.000000000
   [90,]  0.043137255  0.066666667  0.047058824  0.031372549  0.031372549 -0.113725490
   [91,]  0.031372549  0.054901961  0.039215686  0.015686275  0.031372549 -0.160784314
   [92,]  0.000000000  0.003921569 -0.058823529  0.000000000 -0.062745098  0.000000000
   [93,] -0.023529412  0.000000000  0.003921569 -0.007843137 -0.007843137 -0.039215686
   [94,] -0.027450980  0.003921569 -0.019607843  0.000000000 -0.027450980 -0.007843137
   [95,] -0.019607843 -0.019607843 -0.007843137  0.003921569 -0.003921569 -0.023529412
   [96,]  0.011764706 -0.015686275  0.011764706  0.011764706 -0.007843137 -0.023529412
   [97,]  0.137254902  0.192156863  0.188235294  0.164705882  0.117647059  0.196078431
   [98,] -0.047058824 -0.035294118 -0.086274510 -0.070588235 -0.043137255 -0.035294118
   [99,]  0.180392157  0.219607843  0.023529412  0.239215686  0.133333333  0.031372549
  [100,] -0.023529412 -0.015686275 -0.050980392  0.039215686  0.023529412  0.027450980
  [101,]  0.074509804  0.000000000  0.043137255  0.192156863  0.043137255  0.160784314
  [102,]  0.082352941  0.086274510  0.113725490 -0.011764706  0.027450980 -0.670588235
  [103,] -0.349019608 -0.372549020 -0.054901961  0.031372549 -0.082352941  0.035294118
  [104,] -0.007843137 -0.007843137  0.011764706 -0.023529412 -0.031372549 -0.211764706
  [105,]  0.176470588  0.188235294  0.172549020  0.086274510 -0.039215686  0.074509804
  [106,]  0.035294118  0.035294118  0.058823529  0.015686275  0.027450980  0.023529412
  [107,]  0.000000000 -0.003921569  0.011764706  0.019607843  0.000000000  0.011764706
  [108,] -0.007843137  0.015686275  0.015686275 -0.011764706  0.027450980  0.000000000
  [109,] -0.207843137 -0.203921569 -0.184313725  0.003921569  0.023529412 -0.215686275
  [110,]  0.058823529  0.011764706 -0.015686275  0.039215686  0.015686275 -0.023529412
  [111,]  0.003921569  0.011764706  0.000000000  0.000000000  0.011764706  0.007843137
  [112,]  0.000000000  0.035294118 -0.078431373 -0.129411765 -0.086274510 -0.333333333
  [113,] -0.117647059  0.133333333 -0.215686275  0.196078431 -0.294117647  0.156862745
  [114,]  0.243137255  0.015686275 -0.078431373  0.223529412 -0.054901961  0.094117647
  [115,] -0.082352941  0.000000000 -0.078431373 -0.054901961  0.023529412 -0.643137255
  [116,]  0.007843137 -0.011764706  0.000000000 -0.027450980  0.011764706  0.015686275
  [117,] -0.109803922 -0.047058824 -0.035294118 -0.152941176 -0.027450980  0.011764706
  [118,] -0.078431373 -0.066666667 -0.094117647  0.003921569  0.039215686  0.031372549
  [119,] -0.058823529 -0.098039216 -0.031372549  0.000000000  0.027450980  0.007843137
  [120,]  0.015686275 -0.007843137  0.027450980  0.007843137  0.035294118  0.011764706
  [121,]  0.015686275  0.000000000  0.000000000  0.035294118  0.031372549  0.000000000
  [122,]  0.023529412  0.015686275  0.000000000 -0.011764706  0.003921569 -0.019607843
  [123,]  0.007843137  0.011764706 -0.062745098 -0.015686275 -0.039215686 -0.066666667
  [124,]  0.109803922  0.176470588  0.023529412  0.105882353  0.031372549  0.027450980
  [125,]  0.113725490  0.090196078  0.611764706  0.000000000  0.258823529  0.262745098
                 [,7]         [,8]
    [1,] -0.007843137 -0.023529412
    [2,] -0.282352941 -0.156862745
    [3,]  0.066666667  0.000000000
    [4,]  0.082352941  0.011764706
    [5,] -0.113725490 -0.101960784
    [6,] -0.164705882 -0.172549020
    [7,] -0.035294118 -0.035294118
    [8,]  0.239215686  0.239215686
    [9,] -0.054901961 -0.050980392
   [10,]  0.003921569 -0.121568627
   [11,]  0.062745098  0.070588235
   [12,] -0.007843137  0.000000000
   [13,]  0.019607843  0.149019608
   [14,] -0.196078431 -0.043137255
   [15,] -0.172549020 -0.219607843
   [16,]  0.023529412  0.443137255
   [17,]  0.003921569  0.015686275
   [18,] -0.239215686 -0.301960784
   [19,] -0.007843137 -0.011764706
   [20,]  0.011764706  0.043137255
   [21,]  0.086274510  0.058823529
   [22,]  0.062745098 -0.023529412
   [23,]  0.023529412  0.031372549
   [24,]  0.015686275  0.003921569
   [25,] -0.035294118 -0.019607843
   [26,]  0.062745098  0.160784314
   [27,] -0.043137255 -0.105882353
   [28,]  0.031372549  0.058823529
   [29,] -0.027450980 -0.015686275
   [30,] -0.003921569 -0.011764706
   [31,]  0.058823529  0.086274510
   [32,] -0.125490196  0.003921569
   [33,]  0.262745098  0.239215686
   [34,] -0.019607843 -0.007843137
   [35,] -0.235294118 -0.388235294
   [36,]  0.011764706  0.039215686
   [37,] -0.011764706  0.007843137
   [38,]  0.031372549  0.015686275
   [39,] -0.164705882  0.000000000
   [40,] -0.054901961  0.113725490
   [41,] -0.007843137  0.003921569
   [42,] -0.007843137 -0.015686275
   [43,] -0.035294118  0.007843137
   [44,] -0.027450980 -0.050980392
   [45,]  0.050980392 -0.007843137
   [46,] -0.023529412  0.043137255
   [47,] -0.066666667 -0.070588235
   [48,]  0.047058824  0.074509804
   [49,] -0.050980392 -0.023529412
   [50,] -0.211764706 -0.094117647
   [51,] -0.027450980 -0.031372549
   [52,] -0.011764706  0.145098039
   [53,] -0.019607843  0.047058824
   [54,] -0.027450980 -0.015686275
   [55,]  0.047058824 -0.043137255
   [56,] -0.011764706  0.011764706
   [57,] -0.039215686 -0.172549020
   [58,] -0.058823529 -0.007843137
   [59,]  0.047058824  0.007843137
   [60,] -0.047058824 -0.098039216
   [61,] -0.050980392 -0.070588235
   [62,]  0.000000000  0.027450980
   [63,]  0.031372549  0.035294118
   [64,] -0.090196078 -0.133333333
   [65,]  0.121568627  0.160784314
   [66,] -0.019607843 -0.054901961
   [67,] -0.015686275  0.003921569
   [68,]  0.003921569  0.007843137
   [69,]  0.019607843  0.007843137
   [70,]  0.129411765 -0.152941176
   [71,]  0.015686275 -0.035294118
   [72,] -0.145098039 -0.200000000
   [73,]  0.682352941  0.713725490
   [74,]  0.200000000  0.090196078
   [75,]  0.023529412 -0.003921569
   [76,] -0.086274510  0.023529412
   [77,]  0.482352941  0.549019608
   [78,]  0.086274510  0.223529412
   [79,] -0.666666667 -0.690196078
   [80,]  0.149019608  0.129411765
   [81,] -0.070588235 -0.074509804
   [82,]  0.086274510  0.027450980
   [83,]  0.007843137  0.000000000
   [84,]  0.015686275  0.011764706
   [85,]  0.054901961  0.031372549
   [86,]  0.054901961  0.266666667
   [87,] -0.050980392  0.031372549
   [88,] -0.003921569 -0.007843137
   [89,]  0.031372549  0.027450980
   [90,] -0.156862745 -0.156862745
   [91,]  0.011764706  0.000000000
   [92,]  0.000000000  0.000000000
   [93,] -0.023529412 -0.011764706
   [94,] -0.023529412  0.003921569
   [95,] -0.011764706 -0.011764706
   [96,] -0.035294118 -0.023529412
   [97,]  0.184313725  0.145098039
   [98,] -0.058823529 -0.070588235
   [99,]  0.098039216  0.066666667
  [100,]  0.007843137 -0.007843137
  [101,] -0.054901961  0.082352941
  [102,] -0.639215686 -0.529411765
  [103,]  0.003921569 -0.031372549
  [104,] -0.156862745 -0.094117647
  [105,]  0.047058824  0.074509804
  [106,] -0.007843137 -0.058823529
  [107,]  0.007843137  0.035294118
  [108,]  0.000000000  0.000000000
  [109,] -0.129411765 -0.207843137
  [110,] -0.027450980 -0.082352941
  [111,]  0.011764706 -0.011764706
  [112,] -0.466666667 -0.352941176
  [113,] -0.121568627 -0.270588235
  [114,]  0.184313725  0.125490196
  [115,] -0.631372549 -0.352941176
  [116,]  0.070588235  0.019607843
  [117,]  0.090196078  0.003921569
  [118,]  0.066666667 -0.062745098
  [119,]  0.007843137  0.007843137
  [120,]  0.015686275  0.019607843
  [121,]  0.000000000  0.000000000
  [122,] -0.027450980  0.015686275
  [123,] -0.031372549  0.074509804
  [124,] -0.007843137 -0.192156863
  [125,] -0.027450980  0.070588235

 [ reached getOption("max.print") -- omitted 9875 row(s) and 2 matrix slice(s) ]

Step 4: Train a classification model with training images

Call the train model and test model from library.

train.R and test.R should be wrappers for all your model training steps and your classification/prediction steps. + train.R + Input: a path that points to the training set features and responses. + Output: an RData file that contains trained classifiers in the forms of R objects: models/settings/links to external trained configurations. + test.R + Input: a path that points to the test set features. + Input: an R object that contains a trained classifier. + Output: an R object of response predictions on the test set. If there are multiple classifiers under evaluation, there should be multiple sets of label predictions.

source("../lib/train.R")
source("../lib/test.R")

source("../lib/cross_validation.R")
if(run.cv){
  err_cv <- array(dim=c(length(model_values), 2))
  for(k in 1:length(model_values)){
    cat("k=", k, "\n")
    err_cv[k,] <- cv.function(feat_train, label_train, model_values[k], K)
  }
  save(err_cv, file="../output/err_cv.RData")
}

k= 1 

k= 2 

k= 3 

k= 4 

k= 5 

NA

model_best=model_values[1]
if(run.cv){
  model_best <- model_values[which.min(err_cv[,1])]
}
par_best <- list(depth=model_best)

Step 5: Super-resolution for test images

Feed the final training model with the completely holdout testing data. + superResolution.R + Input: a path that points to the folder of low-resolution test images. + Input: a path that points to the folder (empty) of high-resolution test images. + Input: an R object that contains tuned predictors. + Output: construct high-resolution versions for each low-resolution test image.

source("../lib/superResolution.R")
test_dir <- "../data/test_set/" # This will be modified for different data sets.
test_LR_dir <- paste(test_dir, "LR/", sep="")
test_HR_dir <- paste(test_dir, "HR/", sep="")
tm_test=NA
if(run.test){
  load(file="../output/fit_train.RData")
  tm_test <- system.time(superResolution(test_LR_dir, test_HR_dir, fit_train))
}

Summarize Running Time

Prediction performance matters, so does the running times for constructing features and for training the model, especially when the computation resource is limited.

tm_feature_train[1]

user.self 
   84.257